The present invention relates to a method for programming a robot to imitate the motions of a human being or animal under the control of a computer. The present invention further relates to a method for producing a numerical control program for driving a machine tool or similar device in a manner which imitates the motions of a human machine operator.
Recently, industrial robots have come to be used in various fields to perform a wide variety of physical motions which simulate the motions of human beings. Programming a robot involves a process known as teaching. One teaching method is an on-line programming method known as "teaching playback" in which the motions of a human being are directly taught to the robot. Another method is an off-line programming method in which the motions of a human being are simulated by a computer.
In the teaching playback method, a person moves a robot in accordance with a working sequence by manual control. This method is widely adopted because it is easy to perform, but it has a disadvantage in that the operation of the robot must be stopped during the teaching process, thereby lowering the working efficiency of the robot. The off-line programming method is useful when a high working efficiency is required, because the programming can be performed without stopping operation of the robot.
Generally, in the off-line programming method, the contents of tracks and movements scheduled by environmental models are described by a robot programming language. The contents are ascertained by motion simulation and fine adjustment of the contents is later performed at the work site. Thus, the off-line programming method is very complicated. Since the environmental models include geometrical information such as configuration, dimensions, positions, and postures of the workpiece and peripheral equipment, physical parameters such as materials and weights, and technical information such as functions, service and usage, a programmer must schedule the working,sequence by using complex environmental models.
Modern machine tools are frequently controlled by numerical control programs. Recently, in numerical control techniques, systems known as computer-aided engineering (CAE) systems or computer-integrated manufacturing (CIM) systems have been utilized. In these system, design is performed using computer-aided design (CAD) processes and production is performed using computer-aided manufacturing (CAM) processes.
Since numerical control techniques do not specify working requirements, such as the direction of the working operation, the magnitude of the force to be applied to the workpiece, or the working speed, it is impossible with these techniques to achieve the same quality as can be achieved by a skilled craftsman manually controlling a machine tool.
Furthermore, while the formation of numerical control programs is preferably performed by a method having real time response, conventional numerical control programming methods have no real time response because they require the ascertainment of the contents by means of motion simulation and fine adjustment at the work site.
In order to perform high quality manufacturing by means of numerical control, it is necessary to analyze the motions of a human machine operator and form a control program on the basis of the analysis. In order to control a robot to perform the motions of a human being, it is necessary to analyze the motions of a human being and then design motions to be performed by the robot. For both robots and numerical control programs, the motions of an object are observed by a motion analyst and then on the basis of his observations, a curriculum or a program for teaching or producing the same motions is formed. However, because observations are subjective, often based on the intuition of the programmer, they may not give accurate results.
There has recently been proposed a technique wherein the motions of a human being or an animal are analyzed using a computer. See D. I. Miller, "Computer Simulation of Human Motion", in D. W. Grieve et al, (eds), Techniques for the Analysis of Human Motion, Lepus Books, London, 1975; Calvert T. W. et al, "Composition of Multiple Figure Sequences for Dance and Animation", Proceedings CG International '89, pp. 245-254 (1989); Nigel W. J. et al, "Some Methods to Choreograph and Implement Motion in Computer Animation", Proceedings of Computer Animation '89, 125-140 (1989). In analyzing using computer, a motion analyst analyzes the motions depending on his understanding by applying kinematics, which describes the motions only in terms of positions, velocities and accelerations, neglecting the forces and torques responsible. Thus, kinematics analysis can only generate a line picture representing parts constituting a human or animal body, and a three-dimensional model of the human or animal body cannot be displayed realistically on a screen. Accordingly, the motions designed may be unreasonable and also it is difficult to develop a new motion. The same is true when using kinematics to analyze and design the motion of a machine tool. Thus, as with robotic motion, it was difficult to construct a numerical control program using only a computer.
Another method uses dynamics. While kinematics provides motion analysis in terms of positions, velocities and accelerations, dynamics provides the motions of objects based on the relation between movements and forces. If dynamics is applied to the computer analysis of motions, it is possible to generate complex behavior with the analyst's minimal control. However, the computer analysis of motions utilizing dynamics requires data such as the moments of inertia, the centers of gravity, joint friction and muscle/ligament elasticity of the object, which are difficult to measure. Without such data, the computer analysis of motions utilizing dynamics provides unreasonable results similar to the case of using kinematics. Furthermore, it is necessary to solve rather complex dynamics equations. For a human body, an articulated body with 200 degrees of freedom, it is necessary to solve 600 simultaneous differential equations.
In order to display the motions on a screen, the dynamics equations must be solved exactly. Since this method has the computation complexity O(n).sup.4, a function of n.sup.4, where n is the number of segments constituting a human body and forming the minimal units of motion in the motion analysis, large amounts of computations are required and thus takes time and are expensive. Thus, dynamical simulation methods such as that disclosed in Wilhelms J. P. et al., "Using Dynamic Analysis to Animate Articulated Bodies such as Humans and Robots", Proceedings, Graphics Interface '85, 97-104 (1985) have not been accepted, mostly because of its conceptual complexity and computational expensiveness.
A motion analyzing method which reduces the computation complexity to O(n), a function of n, enabled by neglecting rotations about the principal axes, has been proposed. Armstrong W. W. et al, "The Dynamics of Articulated Rigid Bodies for Purposes of Animation", The Visual Computer, 1:231-240 (1985). However, when the rotations of joints about the principal axes cannot be neglected, this method is not applicable. Thus, by using this method, the forces that produce a motion or the forces that are produced by a motion cannot be displayed exactly.
In a publication from the department of the present inventors, Sun, L. et al., "An Architectural Design of a Sports Instruction System", First International Conference and Exhibition on Visual Computing for Defense and Government (VISUDA '89), Paris, France (1989) (Technical Report 89-017), a system for inputting and analyzing human body movement is disclosed. The human body is modeled as a set of articulated segments, each of which is essentially a rigid solid. In this model, the human body is represented by 50 segments. The system takes in a set of measurements taken with an actual human body and converts it into the size data for each segment, e.g., length, width and diameter. When displaying the model, the system approximates each segment with a polygon and performs a smooth shading for better appearance. In addition, the system calculates the center of gravity and the mass for each segment which will be required in the analysis. A video recorder, preferably several video recorders, recording the movement from a plurality of points of view, records the motion of actual humans in a time sequence of video frames. The position of objects in each frame is input to the system to reproduce and analyze the movement. In order to define the position and configuration of the body in space, the position and orientation of each segment are specified. Therefore, for each segment there are six degrees of freedom, three for the position and three for the orientation. Each segment in an articulate body, however, is restricted by attachments to neighboring segments. Therefore, it suffices to specify the position of only one segment (orientation must be specified for all segments), thereby reducing the degrees of freedom considerably. The user specifies the position of each body segment by manipulating this figure using a mouse; the user picks a segment and translates and or rotates it so that the displayed figure assumes the position of the figure on the videotape. The image of a video frame can be superimposed on each window to facilitate this process.
The body movement which has been input into the computer is then analyzed considering two factors: the movement of the center of gravity and the force exerted on each segment of the body. The position vector G of the center of gravity of a human body is computed by the following formulation: ##EQU1## where g.sub.i is the position vector of the center of gravity of a segment i and .DELTA.m.sub.i. Since each segment is assumed to be a uniform rigid body, g.sub.i and .DELTA.m.sub.i can be obtained prior to this calculation from the measurement data.
The center of gravity of each segment is computed in a way similar to the above formulation. Each segment is divided into small wedge-shaped volumes around and along the central axis through the segment, and these volumes are summed up to find the position of the center of gravity.
The problem of computing the force exerted on a human body from its movement is a problem of inverse dynamics. One starts with the sequence of positions and obtains the velocity and the acceleration of each part of the body, from which the force that has caused such motion is to be calculated.
Solving a problem of dynamics is essentially solving differential equations which describe the relationship between mass and force and torque applied to it. A variety of formulations are available for describing the dynamics equation. They all produce the same result expressed in slightly different terms. One of the most familiar formulations is Lagrange's equation: ##EQU2## where L is the Lagrangian given by kinetic energy (T) minus potential energy (P), q.sub.i is the ith coordinate, f is a constraint equation, .lambda. is an undetermined multiplier, and Q.sub.i is the force or torque applied to the ith coordinate.
In a general animation system, the dynamics approach is not yet practical due to the high computational complexity and cost required. The articulated body of the Sun publication model, consisting of 50 segments, requires 300 (=50.times.6) simultaneous equations to describe it. However, the use of known forces, such as the reactive force between a foot and a floor, simplifies the situation. For each segment, we have ##EQU3## where m is the mass of the segment, V.sub.g is the velocity vector of the center of gravity, X.sub.a and X.sub.d are the position vectors of its two ends, a and b, X.sub.a and X.sub.b are the velocity vectors, I is the inertia tensor, .omega..sub.g is the angular velocity vector around the center of gravity, and 1 is the distance between two ends.
Sun proposes letting the left side of the above Lagrange's equation be D. Then, EQU D(X.sub.a, X.sub.b, X.sub.a, X.sub.b, .lambda., F.sub.a, F.sub.b)=0
where F.sub.a and F.sub.b are the forces exerted on a and b, respectively. Each segment has six degrees of freedom, and there are six degrees of freedom, and there are six Lagrange's equations describing it. In the above Lagrange's equations D(X.sub.a,X.sub.b,X.sub.a,X.sub.b, .lambda., F.sub.a,F.sub.b), the three components of F.sub.a (x,y,z) and F.sub.b (x,y,z), and .lambda. are unknown. Therefore, there are seven unknown variables in total in the the system of six equations. If the force in one direction can be obtained through some means (e.g. by measurement), .lambda. and five other force components can be computed by solving these equations.
Sun then supposes that a segment i is connected at an end b.sub.1 to an end a.sub.j of another segment j. By Newton's Third Law of Motion, the force exerted on the segment i at b.sub.i is equal and opposite to the force exerted on the segment j at a.sub.3. By solving the equations for the segment i, one also obtains a partial solution (three of the seven unknown variables) for the segment j as well. By repeating this process, solutions can be found for all segments making up the body.
As the starting point of a series of calculations, Sun suggests using the force exerted on a foot by the floor, which was measured in advance, to calculate the force exerted on the joint of the foot and shin. Then one works upward to head and lower arms through shins, thighs, hip, waist and torso. In the end, one has calculated the force exerted on each segment at its ends. Note that the force calculated here is the sum of internal force and external force. Internal force is exerted by muscles on attached joints. For the segments on which no external force is exerted, the result is equal to just the internal force. On the other hand, for the segments with no internal force, the result is equal to the external force.
While this system of inputting and analyzing human body movement is useful for computerizing the dynamics of videotaped human movement, it does not solve the problem of how to control an industrial robot or machine tool in a manner which simulates the motion of a human body as closely as possible without relying on trial and error or an unworkable magnitude of calculations.